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ABSTRACT 

The Operations Engineering Lab (OEL) at 
JPL has developed a simple, generic toolkit 
to integrate the uplink/downlink processes, 
(often called closing the loop ,), in JPL's 
Multimission Ground Data System. This 
toolkit provides capabilities for integrating 
telemetry verification points with predicted 
spacecraft commands and ground events in 
the Mission Sequence Of Events (SOE) 
document. In the JPL ground data system, 
the uplink processing functions and the 
downlink processing functions are separate 
subsystems that are not well integrated 
because of the nature of planetary missions 
with large one-way light times for spacecraft- 
to-ground communication. Our new closed- 
loop monitoring tool allows an analyst or 
mission controller to view and save uplink 
commands and ground events with their 
corresponding downlinked telemetry values 
regardless of the delay in downlink telemetry 
and without requiring real-time intervention 
by the user. 

An SOE document is a time-ordered list of all 
the planned ground and spacecraft events, 
including all commands, sequence loads," 
ground events, significant mission activities, 
spacecraft status, and resource allocations. 
The SOE document is generated by 
expansion and integration of spacecraft 
sequence files, ground station allocations, 
navigation files, and other ground event files. 
This SOE generation process has been 
automated within the OEL and includes a 


graphical, object-oriented SOE editor and 
real-time viewing tool running under 
X/Motif. The SOE toolkit was used as the 
framework for the integrated implementation. 

The SOE is used by flight engineers to 
coordinate their operations tasks, serving as a 
predict data set in ground operations and 
mission control. The closed-loop SOE toolkit 
allows simple, automated integration of 
predicted uplink events with correlated 
telemetry points in a single SOE document 
for on-screen viewing and archiving. It 
automatically interfaces with existing real- 
time or non real-time sources of information, 
to display actual values from the telemetry 
data stream. 

This toolkit was designed to greatly simplify 
the user’s ability to access and view telemetry 
data, and also provide a means to view this 
data in the context of the commands and 
ground events that are used to interpret it. A 
closed-loop system can prove especially 
useful in small missions with limited 
resources requiring automated monitoring 
tools. This paper will discuss the toolkit 
implementation, including design trade-offs 
and future plans for enhancing the automated 
capabilities. 


INTRODUCTION 

The Operations Engineering Lab (OEL) at 
NASA s Jet Propulsion Laboratory has 
developed a simple, generic toolkit that 
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integrates uplink events with downlink 
telemetry information, (often called closing 
the loop,). This toolkit provides capabilities 
for integrating telemetry verification points 
and ground monitoring information with 
planned spacecraft commands and ground 
events in the Mission Sequence Of Events 
(SOE) schedule. In the existing SOE for 
planetary missions, each spacecraft command 
item has a descriptive text field that contains a 
list of related engineering telemetry 
parameters. These parameters are monitored 
by mission controllers using a data monitor 
tool that processes and displays the downlink 
telemetry stream. However, the relevant 
downlink telemetry for command verification 
may not arrive for hours or even days after 
the commands have been sent because of the 
large one-way light times and limited contact 
periods of some planetary missions. The 
closed-loop system will integrate these tasks 
by interfacing the SOE with the real-time or 
non-real-time telemetry data streams and 
automatically append appropriate channel 
values and limit checks with command and 
ground event items. Our new closed-loop 
monitoring tool allows an analyst or mission 
controller to browse and archive uplink 
commands and ground events with their 
corresponding downlinked telemetry values 
regardless of the delay in downlink telemetry 
and without requiring real-time intervention 
by the user. Figure 1 shows a sample SOE 
with integrated telemetry channel 
information. 

BACKGROUND 

An SOE document is a time-ordered list of all 
the planned ground and spacecraft events, 
including all commands, sequence loads, 
ground events, significant mission activities, 
status and resource allocations. The SOE 
document is generated by the multimission 
control team by expansion and integration of 
sequence and ground files. This SOE 
generation process has been automated and 
includes a graphical, object-oriented SOE 
editor and viewing tool running under 
X/Motif. The SOE is used by the mission 
controllers and spacecraft and instrument 
engineers to coordinate their operations tasks, 
serving as a predict data set in ground 
operations and mission control. 


APPROACH 

The Operations Engineering Lab (OEL) 
proposed a research task to design and 
implement a toolkit that allows simple, 
automated integration of predicted uplink 
events with actual downlink telemetry in a 
single SOE document for viewing and 
archiving. This integrated SOE serves as the 
basis for a closed-loop monitoring toolkit that 
can automatically interface with existing real- 
time or non-real-time sources of information 
and display only selected values from the 
telemetry data stream. 

A significant research effort was in the design 
and implementation of the interprocess 
communication interfaces and interactive 
controls for retrieving and passing 
information from a variety of downlink 
processing applications to the SOE tool. A 
flexible approach was chosen to allow 
phasing of planned future enhancements, 
including expansion of the SOE capabilities 
for automated mission controller logs, 
telemetry logging, system test procedure 
execution, and automated command 
verification. 

During the requirements analysis, the JPL 
Multimission Control Team (MCT) indicated 
that the closed-loop monitoring system 
should include automated log keeping 
capabilities for mission controllers in order to 
include their real-time logs in the sequence of 
events. In the current operations 
environment, the MCT logs real-time 
information on specific uplink and downlink 
events on hand-written forms. The MCT 
cannot meet an electronic logging requirement 
without automation tools - manually typing a 
log report would be impossible during 
intense periods. Although the SOE tool can 
currently be used to enter comments and act 
as a logging tool, the mission controller 
logging requirements were very broad and 
not clearly defined enough to implement 
within an already-developed tool such as the 
SOE editor. Thus, we decided to implement a 
separate tool for automated logging that 
would integrate manual log inputs, predicted 
events input files (such as the SOE), real-time 
broadcast data, and output from other 
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downlink telemetry processing and 
monitoring applications. 

OEL LOGGING TOOL 

The OEL Logging Tool (OLOG) provides 
automated and manual logging of predicted 
and actual mission events in a graphical easy- 
to-use format, The user interface and pull- 
down menus are completely configurable by 
an individual end-user to meet mission- 
specific needs. An initialization file can be 
read on startup that customizes menu options 
and defaults. The OEL Logging Tool is 
shown in Figure 2. 

The OEL Log Tool provides capabilities for 
manual entries to the log, either by allowing a 
user to select items off pre-defined menus or 
entering text manually into the entry area. The 
pre-defined menus and default values can be 
customized for various teams by creating an 
initialization file. The OEL Log Tool is also 
designed to interface in real-time with 
external input sources (downlink telemetry, 
monitor data, interprocess messages, predicts 
and actuals files), thus providing capabilities 
for automated log entries. The tool is 
designed to allow automated and manual log 
entries to occur concurrently with automatic 
ordering based on time tags. The current tool 
implementation includes several mechanisms 
for communicating with downlink processes 
based the MGDS custom Data Transport 
Subsystem (DTS) services. We have also 
implemented a telemetry data processing 
program that captures and processes data 
from the real-time telemetry data stream or 
spooler files, and passes log messages into 
the OEL Log Tool. 

INTERPROCESS COMMUNICATION 

The OEL Log Tool uses the MGDS DTS 
services to implement message passing 
capabilities. The tool connects to a broadcast 
channel or virtual circuit if a channel option is 
chosen. While processing input events, the 
tool continually checks for pending 
messages. 


message passing mechanisms. These 
programs provide a flexible approach to 
process various input data sources, which are 
then reformatted as log messages to be sent to 
the OEL Log Tool. 

• A program that uses a DTS broadcast 
channel or virtual circuit for sending time- 
stamped messages. The OEL Log Tool 
cooperates by receiving and processing 
the messages for input into the log in time 
order. 

• A generic routine that uses the MGDS 
Data Transport functions (DTS) to 
connect to a broadcast channel or virtual 
circuit for sending time-stamped 
messages and a corresponding OEL Log 
Tool routine that creates a broadcast 
channel or virtual circuit to receive 
messages. 

• A program that reads the output from 
an existing real-time Smart Alarm Tool. It 
processes the output and then sends log 
messages to the OEL Log Tool. 

• A program that reads MGDS 
downlinked telemetry data from a virtual 
circuit, real-time broadcast channel, a 
bytestream file, and/or a spooler file 
using the DTS functions. It processes the 
data by parsing telemetry headers and 
checking for selected data types which are 
then extracted, processed, and passed to 
the OEL Log Tool. It uses DTS services 
to send log messages to the OEL Log 
Tool. This program provides a model for 
building additional programs for 
processing data from real-time telemetry 
streams or from data output by other 
applications and then generating log 
messages to be integrated with the predict 
information in the log tool. 

Figure 3 shows the OEL Log Tool and its 
telemetry processing interfaces. 


CLOSED-LOOP SOE TOOL 


Several programs and routines have been The new closed-loop SOE tool is designed to 
written to implement real-time downlink data allow integration of uplink predict events 
processing functions and cooperative with actual downlink telemetry values 
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Ground event and spacecraft command items 
in the SOE have associated telemetry 
parameters that can be used to verify the 
execution of each command or ground event 
or to establish the state of the spacecraft or 
ground system. These parameters are now 
interleaved with the commands and ground 
events in the closed-loop SOE, for easy 
access and viewing by the analyst. The 
closed-loop monitoring system can 
automatically interface with existing real-time 
sources of information, to capture and 
display actual values from the telemetry data 
stream. The graphical SOE viewing tool 
allows a user to highlight events of interest 
with a mouse on the screen and mark them 
for automatic alarm notification in real-time. 
This tool also allows a user to run the SOE in 
demand mode, specifying any time desired, 
and to search or step through the document 
for events of interest. 

The following capabilities have been 
incorporated into the SOE tool to implement 
real-time, closed-loop monitoring capabilities: 

• Real-Time Capabilities'. A scroll bar 
has been added to allow for scrolling 
during real-time viewing. Real-time 
viewing can be accessed and controlled 
from a menu. It is possible to step 
through each event under user control or 
to run the SOE in real-time or in playback 
mode. The real-time control capabilities 
are based on an interrupt algorithm. 

• Channel Objects : A new item format 
has been defined for the SOE to allow 
creation of a telemetry channel object. 
Each channel object is defined by a 
description field and is associated with 
the uplink predict item preceding it. There 
can be multiple channel objects associated 
with a single uplink event. The 
uplink/downlink command/channel 
integration is based on cross-referenced 
information within the telemetry and 
command dictionaries. 

• Channel Options : A new option 

button has been implemented that allows 
a user to optionally display channel 
records in the SOE. 

• Downlink Interface : Interprocess 

communication and data transport 
functions have been implemented to allow 


integration of the downlink telemetry 
stream. 

Figure 4 shows the graphical closed-loop 
SOE tool. 

BENEFITS 

New missions are demanding electronic 
mission controller logs rather than the current 
hand-written reports. Without the automated 
log keeping capabilities in the new OEL Log 
Tool, a controller would spend most of their 
time manually typing in electronic log 
reports. 

The closed-loop SOE and OEL Log Tool 
implementations have laid the groundwork 
for an advanced closed-loop monitoring 
system that can significantly reduce the need 
for operations teams to understand the 
complex set of processing and display tools 
in the existing JPL Multimission Ground 
Data System (MGDS). The current MGDS 
approach requires understanding multiple 
subsystems, and their often subtle interfaces, 
to allow an end-to-end processing of 
downlink data. There is also no current 
method to integrate or compare predict with 
actual values. The closed-loop SOE tool can 
greatly simplify a user's ability to access and 
view telemetry data, and provides a means to 
view this data in the context of the commands 
and predicted values that are used to interpret 
it. In this context, it is expected that 
significant cost savings can be realized from 
the productivity improvements that will be 
realized over the hundreds of current users of 
the SOE document and related uplink tools. 

Some proposed areas of future work include 
expansion of the closed-loop SOE capabilities 
for adaptation to system test, dynamic alarms 
based on tolerances from predicted events, 
automated monitoring of spacecraft and 
ground system configurations at selected 
times, real-time timeline display of the SOE, 
generation of as-flown SOE schedules, and 
automated command verification. An 
advanced closed-loop monitoring system is 
essential to a more automated monitor and 
control system and to significant productivity 
improvements for the smaller missions of the 
future. 
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Figure 1 - Sequence of Events (SOE) Printout 



Figure 4 - Realtime Closed Loop SOE Tool 
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Figure 2 - OEL On-Line Logging Tool (OLOG) 
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Figure 3 - Sample OLOG Printout 
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